Setting up Billing Exports: BigQuery
In this lesson, you will learn about exporting billing data to BigQuery (BQ).
We'll cover the following
Previously, GCP used to also support file export options, but it is now deprecated by GCP.
In this lesson, we will create a sink to export billing data into the BigQuery dataset. This will allow us to query our billing data. For a smaller dataset, this doesn’t make much sense. However, when an organization is large enough to generate more than GBs of billing data, BQ analytics make much more sense to learn the costs and spending patterns.
Exporting billing data#
To enable the export, follow the steps given below.
-
Main menu > Billing
-
Select the primary billing account. Click on Billing export.
-
Select the “Bigquery export” option.
- By default it is disabled you need to enable the billing.
- Click on the “daily cost detail” section “Edit Settings” buttons.
The standard practice is to keep the historical data or bookkeeping data in a separate project not used by development resources. This demo is up to you By now, you should have a fair understanding of creating a project and attaching it to a billing account.
-
GCP will look for a dataset in the BQ. We don’t have any so create one using the “Go To Bigquery” button.
-
This will open up the BQ dashboard.
- In the left pane, click on the project name at the bottom. It will show the “Create Dataset” button on the right side.
-
Once you click the “CREATE DATASET” button, it will open up one form. Fill in the form to create the dataset.
-
We will select “Never” for data expiration as we wanted to store this data forever.
Once we have the dataset ready, go back to the export dashboard and refresh. Click on “Edit settings” and select the project in which the dataset is present.
The dataset will auto-populate this time. Click on the “Save” button.
This will create a table of “Daily cost detail” in the BQ dataset.
The SKU pricing of GCP billing keeps updating. If you want to store the SKU pricing in the same dataset:
- Click on the “Edit Settings” button under the “Pricing” section.
- Select the dataset’s project.
- Enable “Bigquery Data transfer API” if not enabled already.
- Then click Save.
The export will create 2 tables in the dataset. You can get the schema here.
Cost#
For billing exports:
-
Loading data into the designated dataset is free.
-
When exporting and analyzing cloud billing data with BigQuery, the associated cost will depend on the amount of data you stream, store, and query.
You will see the BQ cost structure in the Big Data section. Once we have the billing data, it will be exported to BQ. Since the generated data will be minimal, we will be charged a negligible amount.
Until now, we have been using GCP through UI only, but this is not always how we work in production. In the next chapter, we will look at the GCP command-line interface. We will install Cloud SDK and look at some of the frequently used commands.
Budgets and Alerts
Quiz